We tried to reproduce famous unbiased learning-to-rank results on the large Baidu-ULTR dataset. It turned out that it does not work as intended ... or at least that success depends on how your measure success.
Click models aim to identify and extract biases in search data, but we do not evaluate whether they actually do so. We propose a metric based on relevance annotations that measures the debiasedness of click models, and consequently their usefulness in learning-to-rank from implicit feedback.
We investigated how click models *actually* perform (spoiler: not so great), and whether our offline metrics capture this (spoiler 2: they don't). Here's what it means for ULTR researchers and practitioners.